XG Boost in Mental Health Classification

Fall 2025 Capstone for Data Science

Aabiya Mansoor, Abigail Penza Jackson, Corina Rich, Madelyn Champion (Advisor: Dr. Cohen)

2025-11-28

Introduction

eXtreme Gradient Boosting (XGB) algorithm

  • A supervised machine learning algorithm
  • Modification of Gradient Boosting Framework
  • Ensemble of weak decision trees
  • L1,L2 regularization
  • High performance, speed, scalability
  • Applications in healthcare, education, public health, finance, and engineering.

Introduction

XGBoost for mental disorder classification

  • Using XGBoost on clinical and survey data
  • Classification of individuals into one of four categories: Bipolar I Disorder, Bipolar II Disorder, Major Depressive Disorder, and Normal
  • XGBoosts proves to be a suitable algorithm for the problem

Literature Review

eXtreme Gradient Boosting (XGBoost) in healthcare

  • XGBoost strengths in real world problems handling class imbalance, heterogeneous data types, or non-linear relationships. (Chen and Guestrin 2016)
  • XGBoost combined with DL for breast cancer classification with high reliability. (Liew, Hameed, and Clos 2021)
  • XGBoost with biomarker data to improve depression diagnoses in a large Dutch population dataset.(Sharma and Verbeke 2020)
  • XGBoost to multi-modal datasets to predict self-harm in young adults.(Xu et al. 2024)
  • A hybrid algorithm (XGBoost-HOA) to classify depression, anxiety, and stress
  • Dual XGBoost models applied to distinguish between deficit and non-deficit schizophrenia subtypes using fMRI features.(Zhang, Jia, and Shang 2022)
  • XGBoost compared with linear regression in predicting depression among refugee children.(Saleh et al. 2024)

Literature Review

XGBoost limitations for imbalance data

Literature Review

XGBoost in Education

XGBoost in Public health

  • XGBoost used to predict daily COVID-19 cases in the United States.(Fang et al. 2022)
  • A hybrid model integrating XGBoost, Random Forest, and Antlion Optimization used to predict infectious disease outbreaks.(Sivakumar and Elangovan 2023)

Literature Review

XGBoost in financial sector

  • XGBoost model’s ability to rank feature importance and prevent overfitting using regularization made it a top performer in credit risk prediction tasks(Li et al. 2020)
  • XGBoost to forecast volatility in the U.S. stock market, identifying the Economic Policy Uncertainty Index as a critical predictor.(Fomunyam 2023)

XGBoost In pharmaceutical research

Literature Review

XGBoost in educational diagnostics

XGBoost in Sports Analytics

Methods: Extreme Gradient Boosting (XGBoost)

  • Ensemble of weak decision trees
  • Trained sequentially, each tree corrects previous errors
  • Captures non-linear relationships within the 17-variable dataset
  • Uses gradient descent to minimize loss
  • Includes regularization to prevent overfitting

Methods: XGBoost Formulas

  • Loss term: Measures prediction error
  • Regularization term: Penalizes model complexity
  • Ensures balance between accuracy and generalization

\[ \text{Obj} = \sum_{i=1}^{n} L(y_i, \hat{y}_i) + \sum_{k=1}^{K} \Omega(f_k) \]

Data Exploration and Visualization

  • Sourced from Kaggle (2023 clinical records)
  • 120 individuals assessed for mental disorders -Labels: -Normal -Bipolar Type I -Bipolar Type II -Depression
  • 17 diagnostic variables
  • Mix of clinical, behavioral, demographic factors

Data Exploration and Visualization

  • Examined class distribution
  • Identified potential imbalance or outliers
  • Explored correlations between features

Modeling and Results

Data Preprocessing

To facilitate analysis, all categorical variables in the dataset were transformed into numerical or grouped categorical formats in Excel. The transformations were applied as follows:

  • Ordinal responses (Sadness, Euphoric, Exhausted) recoded:
    • Seldom=1, Sometimes=2, Usually=3, Most-Often=4
  • Binary items (Suicidal Thoughts, Mood Swing):
    • Yes=1, No=0
  • 1–10 rating scales (Sexual Activity, Optimism) grouped:
    • 1–3 → Cat 1, 4–6 → Cat 2, 7–9 → Cat 3

Modeling and Results

Hyperparameter Optimization

  • The dataset was randomly partitioned into an 80/20 training–testing split.
    • This split resulted in 96 patient surveys used for model training and 24 surveys reserved for independent testing.
    • The predictive target for all models was the expert clinical diagnosis
  • An XGBoost model specification was created using the tidymodels framework where all major hyperparameters were set to be tuned including: - the number of trees - tree depth - learning rate - loss reduction, - minimum node size - subsampling rate - number of predictors considered at each split

Modeling and Results

Hyperparameter Optimization

  • Five-fold stratified cross-validation was used for hyperparameter tuning.
    • Where the training dataset was partitioned into five equally sized folds
    • For each tuning iteration:
      • Models were trained on four folds
      • Validated on the remaining fold
      • Which ensured stable and unbiased performance estimates

Five Fold Validation(singh2024crossvalidation_image?)

Modeling and Results

Hyperparameter Optimization

  • The final hyperparameter grid consists of 30 candidate hyperparameter combination which was constructed using a space-filling design.
    • Model performance was assessed using accuracy and the area under the ROC curve (ROC AUC).
    • Visualizing the tuning results revealed the ROC AUC versus number of trees plot indicated that the optimal tree depth was approximately 2, suggesting that relatively shallow trees generalized best for this dataset.
    • Model performance increased steadily with the number of boosting iterations, with the highest ROC AUC occurring at around 625 trees. At this configuration, the model achieved an accuracy exceeding 0.80 and a ROC AUC greater than 0.90, indicating strong discriminatory performance.

VisulizationTuningResults

Modeling and Results

Hyperparameter Optimization

  • The select_best() function from the tidymodels framework was used to systematically extract the hyperparameter combination that achieved the highest ROC AUC across all resampled evaluations.

  • The resulting optimal configuration included 3 predictors (mtry = 3), 627 boosting iterations, a minimum node size (min_n) of 2, a learning rate of 0.16501, a loss-reduction parameter of 0.0034, and a subsample proportion of 0.5655.

    • This combination represents the tuning grid entry labeled pre0_mod06_post0, which achieved the best balance between model complexity and predictive performance during cross-validation.

BestParams

Modeling and Results

Model Evaluation & Performance

Prediction

The model’s predictions on the held-out test set showed strong agreement with the expert diagnoses. Specifically:

  • Bipolar Type-1: 4 of 6 cases were correctly predicted (66.7%)
  • Bipolar Type-2: 5 of 7 cases were correctly predicted (71.4%)
  • Depression: 6 of 7 cases were correctly predicted (85.7%)
  • Normal: 4 of 6 cases were correctly predicted (66.7%)

Modeling and Results

Model Evaluation & Performance

The XGBoost multiclass classification model demonstrates moderate overall predictive performance, with room for improvement but also several encouraging indicators of discriminative ability. - Accuracy: 0.654
An accuracy of 65.4% suggests that the model correctly classifies roughly two-thirds of the observations. - Kappa: 0.534
The Kappa value of 0.534 reflects moderate agreement between the model’s predictions and the true class labels after accounting for chance. - ROC AUC (macro-weighted): 0.927
The macro-weighted ROC AUC of 0.927 suggests that the model does a strong job of ranking observations correctly across all classes, even if it occasionally makes incorrect final class assignments.

Taken together, the metrics indicate that the model has strong underlying discriminative power (high AUC) but moderate final classification performance (accuracy and kappa).

Modeling and Results

Feature Importance & Interepretability

Modeling and Results

Sensitvity & Robustness Analysis

Modeling and Results

Summary of Modeling Results

Conclusion: Study Purpose

  • Evaluate effectiveness of XGBoost for predicting mental health outcomes.
  • Develop, tune, and validate the model.
  • Identify highest‑predictive features and assess ability to capture non‑linear relationships.
  • Model showed strong accuracy and generalizability.

Conclusion: Key Findings

  • Top predictors: stress indicators, behavioral patterns, wellness factors.
  • High feature importance suggests relevance to mental‑health risk.
  • Model effectively distinguished between classes.
  • Stable performance across tuning configurations with minimal overfitting.

Conclusion: Implications

  • XGBoost shows potential for early warning mental‑health monitoring.
  • Useful for identifying at‑risk individuals based on survey data.
  • Self‑reported data may introduce bias—interpret with care.

Conclusion: Future Work

  • Use larger and more diverse datasets.
  • Incorporate behavioral or time‑based variables.
  • Compare with Random Forest, LightGBM, neural networks.
  • Test in real‑world settings.
  • Address fairness and ethical use of health data.

Conclusion: Overview

  • XGBoost is effective for survey‑based mental‑health prediction.
  • Provides fast and interpretable insights alongside traditional assessments.
  • Predictive modeling could expand tools for mental‑health research and practice.

References

Chen, Tianqi, and Carlos Guestrin. 2016. “XGBoost: A Scalable Tree Boosting System.” In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–94. New York, NY, USA: ACM; Association for Computing Machinery. https://doi.org/10.1145/2939672.2939785.
Fang, Zhiyuan, Shuang Yang, Chun Lv, et al. 2022. “Application of a Data-Driven XGBoost Model for the Prediction of COVID-19 in the USA: A Time-Series Study.” BMJ Open 12 (7): e056685. https://doi.org/10.1136/bmjopen-2021-056685.
Fomunyam, R. A. 2023. “The Impact of the u.s. Macroeconomic Variables on the CBOE VIX Index.” Journal of Economics and Finance 47 (1): 77–94. https://www.proquest.com/docview/2642416749.
Hakkal, Soukaina, and Ayoub Ait Lahcen. 2024. “XGBoost to Enhance Learner Performance Prediction.” Computers and Education: Artificial Intelligence 7: 100254. https://doi.org/10.1016/j.caeai.2024.100254.
Hu, Ting, and Ting Song. 2019. “Research on XGBoost Academic Forecasting and Analysis Modelling.” Journal of Physics: Conference Series 1324 (1): 012091. https://doi.org/10.1088/1742-6596/1324/1/012091.
Li, H., Y. Cao, S. Li, J. Zhao, and Y. Sun. 2020. XGBoost Model and Its Application to Personal Credit Evaluation.” IEEE Intelligent Systems 35 (3): 52–61. https://doi.org/10.1109/MIS.2020.2972533.
Liew, Xin Yu, Nazia Hameed, and Jeremie Clos. 2021. “An Investigation of XGBoost-Based Algorithm for Breast Cancer Classification.” Machine Learning with Applications 6: 100154. https://doi.org/10.1016/j.mlwa.2021.100154.
Nikolaidis, P. T., Beat Knechtle, and other co-authors. 2023. “Analysis of the 10-Day Ultra-Marathon Using a Predictive XGBoost Model.” Open Sports Sciences Journal 16. https://uwf-flvc.primo.exlibrisgroup.com/discovery/fulldisplay?docid=cdi_doaj_primary_oai_doaj_org_article_986cc6e5973948ed919ab7ac5176113a.
Saleh, M., E. Amona, M. Kuttikat, I. Sahoo, D. Chan, J. Murphy, and M. Lund. 2024. “Child Mental Health Predictors Among Camp Tamil Refugees: Utilizing Linear and XGBOOST Models.” PLoS ONE 19 (9): e0303632. https://doi.org/10.1371/journal.pone.0303632.
Sharma, Anjali, and Wouter J. M. I. Verbeke. 2020. “Improving Diagnosis of Depression with XGBOOST Machine Learning Model and a Large Biomarkers Dutch Dataset (n = 11,081).” Frontiers in Big Data 3: 15. https://doi.org/10.3389/fdata.2020.00015.
Sivakumar, R., and S. Elangovan. 2023. “Prediction of Seasonal Infectious Diseases Based on Hybrid Machine Learning Approach.” International Journal of Health Sciences 7 (2): 1958–69. https://research.ebsco.com/c/imx7og/viewer/pdf/ff4a3en7vb.
Su, Wenjie, Fei Jiang, Chen Shi, Dapeng Wu, Lihua Liu, Shu Li, Ying Yuan, and Jie Shi. 2023. “An XGBoost-Based Knowledge Tracing Model.” International Journal of Computational Intelligence Systems. https://doi.org/10.1007/s44196-023-00192-y.
Wiens, Mark, April Verone-Boyle, Nate Henscheid, J. T. Podichetty, and John Burton. 2025. “A Tutorial and Use Case Example of the eXtreme Gradient Boosting (XGBoost) Artificial Intelligence Algorithm for Drug Development Applications.” Clinical and Translational Science 18: e70172. https://doi.org/10.1111/cts.70172.
Xu, Xiao-Ming, Yang S. Liu, Su Hong, Chuan Liu, Jun Cao, Xiao-Rong Chen, Zhen Lv, et al. 2024. “The Prediction of Self-Harm Behaviors in Young Adults with Multi-Modal Data: An XGBoost Approach.” Journal of Affective Disorders Reports 16: 100723. https://doi.org/10.1016/j.jadr.2024.100723.
Zhang, Ping, Yibing Jia, and Yanan Shang. 2022. “Research and Application of XGBoost in Imbalanced Data.” International Journal of Distributed Sensor Networks 18 (6). https://doi.org/10.1177/15501329221106935.